The present technology relates to an image processing apparatus, an image processing method, and a program, and particularly to an image processing apparatus, an image processing method, and a program that are capable of obtaining an image signal of each color component from an output from an image sensor with a color filter array that is configured from multiple color components, without degrading image quality when performing image processing for resolution conversion.
In recent years, there has been an increasing demand for an improvement in image resolution, such as so-called 4 k2 k or 8 k4 k. However, for example, even though the size of an image sensor may be increased, it is difficult to realize sufficient improvement in resolution in terms of the size, the weight, or the like of an optical system. Furthermore, it is also difficult to increase a frame rate in terms of sensitivity of the image sensor.
Because of this, a technology has been pursued in which resolution is improved without decreasing an area of a cell of the image sensor to more than a predetermined extent. For example, a technology has been pursued in which a pixel of an output image is generated and the output image with a high resolution is obtained by performing image processing on an input image with a low resolution.
Furthermore, there are mainly two kinds of imaging apparatuses that use the image sensor. One is a one-chip apparatus (hereinafter referred to as a one-chip camera) that uses one image sensor. The other is a three-chip apparatus (hereinafter referred to as a three-chip camera) that uses three image sensors.
In the three-chip camera, for example, three image sensors are used, one for an R signal, one for a G signal, and one for a B signal, and thus three primary color signals are obtained by the three image sensors. Then, a color image signal that is generated from the three primary color signals are stored in a recording medium.
In the one-chip camera, one image sensor is used in which a color coding filter made from a color filter array assigned to every one pixel is installed in the front, and a signal of a color component that is color-coded by the color coding filter is obtained for every one pixel. As the color filter array that makes up the color coding filter, for example, primary color filter arrays for red (R), green (G), and blue (B) or complementary filter arrays for yellow (Ye), cyan (Cy), and magenta (Ma) are used. Then, in the one-chip camera, a signal of one color component is obtained for one pixel by the image sensor, a color signal other than the signal of the color component retained by each pixel is generated by performing linear interpolation processing, and thus an image close to an image that is obtained by the three-chip camera is obtained. In a video camera, a one-chip method is employed for miniaturization and weight saving.
As the color filter array that makes up the color coding filter, the color filter array in a Bayer layout is used most of the time. In the Bayer layout, G color filters are arranged in a checkered pattern and R's and B's are alternately arranged in every line on the remaining portion.
In this case, in the image sensor, from each pixel in which a filter for one color among three primary colors, R, G, and B is arranged, only an image signal corresponding to such a filter color is output. That is, from the pixel in which an R color filter is arranged, an image signal of an R component is output, but image signals for a G component and a B component are not output. In the same manner, from a G pixel, only an image signal of the G component is output and the image signals for the R component and the B component are not output. From a B pixel, only the image signal of the B component is output and the image signals for the R component and the G component are not output.
However, the image signals of the R component, the G component and the B component are necessary at the time of the processing of the signal of each pixel on the downstream side of the image processing. Accordingly, in the technology in the related art, the image signal of n×m R pixels, the image signal of n×m G pixels, and the image signal of n×m B pixels are obtained, by their respective interpolation calculations, from an output from the image sensor that is configured from n×m (n and m are positive integers) pixels, and are output to the downstream side.
Furthermore, a technology is proposed in which the image signal of 2n×2m R pixels is obtained, by the interpolation calculation, from the image signal of n×m R pixels, the image signal of 2n×2m G pixels is obtained, by the interpolation calculation, from the image signal of n×m G pixels, and the image signal of 2n×2m B pixels is obtained, by the interpolation calculation, from the image signal of n×m B pixels (for example, refer to Japanese Unexamined Patent Application Publication No. 2000-341705).
In Japanese Unexamined Patent Application Publication No. 2000-341705, pixel values for the pixel corresponding to an observation pixel and for the vicinity thereof are set to be variables in an input image, and the pixel value for the observation pixel of an output pixel is predicted by a product and sum calculation that uses a coefficient that is obtained by prior learning. By doing this, the three primary color signals can be generated from an output from the image sensor of the one-chip camera, and an image signal with pixel density four times that of an original image can be generated.